Yeah just be very vague and non-committal. Don't get aggressive but stand your ground. Make no apologies, and don't act like your life is anything other than the best it has ever been. When I quit service, I was only regularly putting in 1-3 hrs, 0 everything else. Zero comments and the meetings, no "privileges". After 3 months 2 elders from my FS group came by.
I didn't say anything about why, just said I didn't like going out anymore. When asked why, just pull a Jesus and answer a question with a question. "Is it voluntary, or compulsory?"
Let them go through their guilt, fear and obligation routine (GFO), smile and nod. Thank them for their time and interest, you have their numbers when you need them. Decline all invitations to come out with them. Feed them some coffee and muffins, talk about the NHL and send them on their way. I haven't heard a peep since, and that was November 2012. Now on the group list it just has my wife's name, I'm not included at all.
Also, conviently they will likely stop giving you talks and any other privileges since you aren't exemplary. But seriously, dude, you NEED to drop those hours if you want to stop going out altogether. Don't make it such a surprise! Zero mags, zero everything except for a token hour or two until you are ready for the big donut.